Aws fastpath deploy #240
Open
Aws fastpath deploy #240
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This server was running out of memory trying to bring up the fastpath docker container. I changed the instance type from one with 0.5gb of ram to one with 1gb of ram
If you don't the first time you run this notebook it will crash with "permission required" whenever you try to run a docker command
Terraform Run Output 🤖Format and Style 🖌
|
| Pusher | @LDiazN |
| Action | pull_request |
| Environment | dev |
| Workflow | .github/workflows/check_terraform.yml |
| Last updated | Tue, 22 Jul 2025 07:33:04 GMT |
Ansible Run Output 🤖Ansible Playbook Recap 🔍Ansible playbook output 📖
|
| Pusher | @LDiazN |
| Action | pull_request |
| Working Directory | |
| Workflow | .github/workflows/check_ansible.yml |
| Last updated | Tue, 22 Jul 2025 07:33:26 GMT |
hellais
reviewed
Jun 27, 2025
| dest: "/opt/{{fastpath_user}}/backend" | ||
| # TODO Change to `master` when https://github.com/ooni/backend/pull/935 is merged | ||
| version: support-deploying-fastpath-as-docker-container | ||
| force: yes |
There was a problem hiding this comment.
It would be ideal if we did this using the same pattern as the reset of the services. That would be building the docker image upon each commit in backend automatically and then we do the deploy using the tag.
It otherwise makes it harder to test something in dev and then deploy it inside of prod.
What we would ideally like to do is make a build, tag a version, test it in dev and then when the test passes we can deploy it to prod.
Can you make changes to run this as part of the CD pipeline?
|
We are also missing another piece to get fastpath fully working and that is handling measurement upload. It's necessary to extract the api-uploader systemd unit and component over to fastpath (see: https://github.com/ooni/backend/blob/master/api/debian/ooni-api-uploader.timer). Other relevant docs are for these pieces are here: https://docs.ooni.org/backend/systemd-timers/#ooni-api-uploader-timer I guess we have 2 options on how to handle this:
|
|
I will probably go with 2, but I will address it in a follow up PR with that and setting up the access to s3 and clickhouse |
hellais
reviewed
Jul 17, 2025
| @@ -0,0 +1,9 @@ | |||
| $ANSIBLE_VAULT;1.1;AES256 | |||
There was a problem hiding this comment.
What are we storing in the vault? Can we put this into aws_ssm instead?
hellais
reviewed
Jul 17, 2025
|
|
||
| # Fastpath settings | ||
| # TODO Update this to the actual clickhouse host when we have migrated it | ||
| clickhouse_url: "clickhouse://default:default@clickhouse-server:9000" |
There was a problem hiding this comment.
We have migrated it, should we then update it?
There was a problem hiding this comment.
Yes, actually the fastpath is taking configurations from its the ansible/host_vars/fastpath.dev.ooni.io/vars.yml file but I will update the default one to be the same
hellais
reviewed
Jul 17, 2025
| - name: Ensure fastpath is running | ||
| community.docker.docker_container: | ||
| name: fastpath | ||
| image: ooni/fastpath:latest |
There was a problem hiding this comment.
Can we place a specific version tag in here and make it as variable so we ensure we have the correct tested version?
There was a problem hiding this comment.
Yes, I will this one that has the changes with the fastpath writing to disk:
https://hub.docker.com/layers/ooni/fastpath/v0.87/images/sha256-60b2d4e55fa4ba0a9aeb1e821e901e49f78141619eefa6ce45fd4ef251145ee0
hellais
reviewed
Jul 17, 2025
| - name: Install Statsd | ||
| tags: uploader | ||
| apt: | ||
| name: python3-statsd |
There was a problem hiding this comment.
It's probably cleaner to use the pkg list here instead, like so:
- name: Install dependencies for uploader
tags: uploader
apt:
pkg:
- python3-statsd
- python3-boto3
... etc
hellais
reviewed
Jul 17, 2025
| [DEFAULT] | ||
| # arn:aws:iam::676739448697:user/ooni-pipeline, AWS: OONI Open Data | ||
| aws_access_key_id = AKIAJURD7T4DTN5JMJ5Q | ||
| aws_secret_access_key = {{ s3_ooni_open_data_access_key }} |
There was a problem hiding this comment.
I think this might have been placed in the vault. I suggest we move it instead into parameter store
There was a problem hiding this comment.
I will do that, I was using that pattern because it was used in other places for the same key:
Should we update those places as well?
hellais
reviewed
Jul 17, 2025
| f"raw/{tstamp[:8]}/{tstamp[8:10]}/{cc}/{testname}/{jsonlf.name}" | ||
| ) | ||
| if conf.get("run_mode", "") == "DESTROY_DATA": | ||
| log.info("Testbed mode: Destroying postcans!") |
There was a problem hiding this comment.
This mode of operation scares me a bit. I would suggest we maybe remove it to avoid mistakes.
hellais
reviewed
Jul 17, 2025
tf/environments/dev/main.tf
Outdated
| from_port = 80, | ||
| to_port = 80, | ||
| protocol = "tcp", | ||
| cidr_blocks = ["0.0.0.0/0"], |
There was a problem hiding this comment.
Does it actually need permission to be accessed on port 80 from anywhere on the internet?
There was a problem hiding this comment.
not actually, I will remove it
hellais
reviewed
Jul 17, 2025
| } | ||
|
|
||
| module "fastpath_builder" { | ||
| source = "../../modules/ooni_docker_build" |
There was a problem hiding this comment.
Do we really need a new module for this? I see that the only thing that changes between ooni_docker_build and ooniapi_service_deployer/ is that we are skipping the final Deploy stage. Could we just pass that as an optional flag to it and avoid duplicating all the module code?
There was a problem hiding this comment.
If it's too tricky though we can skip it
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR will add an ansible role to deploy the fastpath as an EC2 isntance using Docker.
The ansible role will:
Known issues
For now this setup has an issue with the Docker installation where the docker role will try to reboot the SSH connection to apply docker group settings, but this seems to crash the next command:
There's also another issue where the Docker deployment fails because it can't create a network for the docker compose. This is fixed by manually rebooting the docker daemon:
systemctl restart docker(see https://stackoverflow.com/questions/54380847/failed-to-setup-ip-tables-unable-to-enable-nat-rule)To solve this issue we have to ensure that docker is rebooted before starting the docker compose
This should be solved by
devops/ansible/roles/fastpath/tasks/main.yml
Line 45 in 4b148c6
Update
After the last review I added the following features:
closes #239